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ABSTRACT 


The study used activity systems analysis to characterize the processes, dynamics, and tensions in the social practices 
developed by a class of academic low-achievers in a knowledge-building environment augmented by analytics-supported 
reflective assessment. A class of 37 Grade 9 low-achievers and an experienced teacher participated this study. Classroom 
data sources including classroom videos and observations, student artifacts, student interviews, and videos of 
reflective-assessment sessions were collected and analyzed. The findings indicated that staging tasks, frequent and 
opportunistic knowledge-building talks, the framing of discourse improvement as collective responsibility, and the 
acquisition of new skills and interests served as new tools mediating the students’ activities. Our findings have important 
implications for the design of technology-rich environments as metacognitive tools to support low achievers. 
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1. INTRODUCTION 


Understanding how low-achievers develop social practices by which they gradually generate creative ideas is 
critical for low-achievers’ continuous development. This study used activity systems analysis (ASA), 
developed by (Yamagata-Lynch, 2010), to map the development of, and the dynamics and tensions in, the 
social practices in which a class of low-achievers collectively advance their ideas in a community context. 
This study was part of a program that examined the design, process, and effects of knowledge building 
environments augmented by analytics-supported reflective assessment on low-achievers. The study addressed 
the following research questions: 

1) What is the nature of social practices from a cultural historical activity theory (CHAT) perspective? 

2) What characterizes the dynamics of the social practices from a CHAT perspective, and how do the 
dynamics drive the development of the social practices? 


2. REFLECTIVE ASSESSMENT AND LOW-ACHIEVING STUDENTS 


In reflective assessment, students use their agency to reflect on a set of criteria/principles or learning goals. 
They produce their own feedback based on continuous assessment of the inquiry process and product, and 
improve their ongoing learning by attacking broader problems and continuously creating knowledge 
(Scardmalia, 2002; White & Frederiksen, 1998). Reflective assessment is collaborative: not everyone in the 
community needs to be highly metacognitive for the process to be effective, and students can scaffold others’ 
metacognitive development through modeling. 
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Research in the learning sciences suggests that students of various achievement levels can participate in 
and benefit from reflective assessment. For example, White and Frederiksen (1998) found that reflective 
assessments carried out by middle-school students positively affected their performance in a 
scientific-inquiry test and a physics test, and that the process was particularly beneficial for low-achieving 
students. Toth et al. (2002) reported a similar intervention in which students were guided by a reflective 
assessment rubric during the inquiry, and found that reflective assessment typically had a positive effect on 
learning. Herrenkohl et al. (2011) found that the use of self- and peer-assessment tasks in an interactive 
online environment, accompanied by assessment criteria and useful tools such as software advisors, can 
scaffold students’ conceptual understanding in science. However, there has been relatively little research into 
how we might assist low-achieving students to carry out this assessment as active agents to improve their 
collaborative inquiry. 

This study characterized and mapped the developmental process of social practices created by 
low-achieving students in a knowledge-building environment augmented by reflective assessment. In this 
study, we provided students an assessment tool, the Knowledge Connections Analyzer (KCA) (Yang et al., 
2016) to support reflective assessment. The KCA is a Web-based analytics tool designed to be used by 
secondary-school students to reflect on their online work in a Knowledge Forum. It queries the Knowledge 
Forum database to collect information on four intuitive questions related to knowledge building. For a full 
description, see Yang et al. (2016). 


3. METHODS 


3.1 Research Context and Participants 


The study was conducted at a Band-3 school in Hong Kong, which is at the 10th percentile. Students at such 
schools tend to be weaker in Chinese and English, communication, and critical thinking than their 
counterparts in Band-1 and Band-2 schools, and have poorer motivation, self-identity, self-image, and 
metacognitive skills. The participants were 37 9" graders from a class taking a visual-arts course. They 
studied the topic of “Art appreciation” over five months in weekly one-hour lessons. Their teacher had 
previously found the students to be more motivated by knowledge building, a promising student-centered 
approach, than by other teaching methods. The teacher had considerable experience of teaching the visual 
arts, had taken a postgraduate course on knowledge building, and had used knowledge building for 
approximately eight years. 


3.2 Pedagogical Design 


The teacher used a three-phase pedagogical process to familiarize the students with knowledge building, as 
described in detail by Yang et al. (2016): (1) Creating a collaborative culture and helping students to build 
skills in inquiry, collaboration, and metacognition, (2) Deepening problem-centered inquiry in the 
Knowledge Forum, an online learning environment that supported students’ online discourse, and (3) Using 
analytics-supported reflective assessment to foster deep domain understanding and metacognitive skills. 


3.3 Data Sources and Analysis 


3.3.1 Data Sources 


In this study, we collected the following interrelated sets of qualitative data: 

Classroom observations. We conducted the observations while the participants were engaged in activities 
related to knowledge building. The data comprised photographs and field notes related to some of the 
lessons, and video recordings of 17 one-hour lessons. 

Video recordings of reflective-assessment sessions. We video recorded the students’ reflective-assessment 
activities while on KF, in class, and during after-class group sessions, including their interpretations of and 
reflections on the data and plans for their KF discourse. 
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Interviews. We conducted semi-structured interviews to obtain information about the students’ 
experiences of KCA. Some of the interviews were conducted individually, and others in groups of 2-3 
students. Most of the interviews were conducted informally in the computer laboratory in which the students 
used the KCA during class, and each interview lasted for 20-30 minutes. The interview questions 
corresponded to the four questions in the KCA to obtain information about students’ use of the KCA to 
support reflective assessment. The interviews were audio recorded. 


3.3.2 Data Analysis 


We used ASA to characterize the social practices that arose from the KCA-aided reflective assessment during 
knowledge building. ASA has been used in several studies to identify the systemic contradictions and 
tensions that shape developments in educational settings (Barab et al., 2002; Kim, 2011). The ASA was 
conducted in two main stages: identifying codes and analyzing activity systems. 

Code identification. We began data analysis using the constant comparative method (Strauss & Corbin, 
1998). Analyses were conducted to explore the potential determinants of students’ productive and 
unproductive use of the KCA to conduct reflective assessment, and to identify the elements and ongoing 
activities that supported the transformation of newly introduced artifacts into cultural tools. The multifaceted 
analysis results of students’ online discourse (Yang et al., 2016) provided the basis for the qualitative ASA. 
The study began by examining video-recordings of reflective assessment sessions to identify productive and 
unproductive uses of the KCA. Next, the study analyzed the potential of reflective assessment to increase 
students’ focus on the key learning goals of knowledge building. The study further systematically analyzed 
the remaining sources of data to cast light on the students’ perceptions of the effectiveness of the KCA in 
supporting reflective assessment, their experiences of using the KCA to conduct reflective assessment, and 
the activities and other elements that assisted or hindered the students’ productive use of the KCA. The goal 
of this analysis was to report on thematic findings on the participants’ engagement in KCA-afforded 
reflective assessment and various activities and other factors that made their use of the KCA productive or 
unproductive. 

Identifying activity systems. This task involved three main steps: (a) developing thick descriptions of the 
participants’ experiences in narrative format, (b) conducting ASA of the narratives, and (c) identifying 
substantiated findings (Yamagata-Lynch, 2010). Before drafting narratives of the students’ experiences of 
self-directed reflective assessment, we identified substantiated stories from the data. “Work done by the 
class” was viewed as an activity system subject to ASA, and Engestrém’s (1987) triangle was used to 
describe the component structure of each instance of this activity. Identifying activity systems was an 
iterative process that involved multiple stages of revision, during which the drafted narratives were also 
modified. After finalizing the narratives and the ASA, we began identifying substantiated findings. 

Validity of ASA. Trustworthiness was enhanced through consistent observation, methodological 
triangulation, and a rigorous coding process. We observed the participants consistently for approximately 
five months. At the same time, we obtained necessary information about the students, the teacher, methods of 
instruction, and curriculum by observing the teacher’s work in other classes, and drawing on our six-year 
research relationship with the teacher. We conducted methodological triangulation using multiple sources of 
data: observation, students’ artifacts, interviews, and questionnaires. 


4. RESULTS 


4.1 Results of ASA 


We constructed three activity systems to describe and explain the development of the reflective-assessment 
activities and the interaction of these activities (a) before the KCA student activity, (b) during the KCA 
student activity, and (c) immediately after the KCA student activity. The activity systems are shown in 
Figures 1-3. 

Activity system A (Figure 1) comprised the students’ knowledge-building activities before using the 
KCA to carry out reflective assessment. In this activity system, a variety of meditational tools enabled the 
students to advance their knowledge, develop skills in summarization, collaboration, and metacognitive 
reflection, and develop a collaborative knowledge building culture. For example, the construction of a 
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knowledge-building wall helped low-achievers to make their ideas visible and public and created a 
collaborative culture conductive to subsequent productive knowledge building. 

These tools motivated the students to invest much more effort in developing both face-to-face and online 
knowledge-building discourse, and to some extent improved their understanding of knowledge building as a 
collective enterprise focusing on idea improvement. However, the students still lacked motivation and 
showed weak inquiry, collaborative, and metacognitive skills, which led to tension (a): the need to attain an 
object despite lacking motivation and skills. In addition, the resource limitation that directed students’ 
attention from individual note-writing to collective knowledge advancement, and urged students to 
continuously review, reflect on, and further synthesize/rise above community ideas, brought about tension 
(b): attaining the object with limited resources. Moreover, the traditional teacher-centered school culture and 
the associated norms for participation undermined the teacher’s efforts to foster knowledge building and the 
students’ efforts to carry out knowledge building, which brought about tension (c): the need to attain an 
object within a traditional teacher-centered school culture and the norms for participation in that culture. 
These tensions placed the student participants involved in activity system A in _ contradictory 
knowledge-building situations, and provided minimal support for their attempts to collaboratively advance 
their discourse. The forces created by these conditions militated against the subjects’ efforts to attain the 
object. 


Tools 
Periodic tasks promoting collaboration and participation 
Collaborative conceptmaps 
Knowledge wall 
Prompt sheets for weekly reflections and paper portfolios 
Knowledge Forum® and integrated assessment tools 
Knowledge-building talks promoting reflection 


Subjects Outcome 
Unmotivated students Object Discouragement and satisfaction 
with limited skills Collaboratively ——> Creation of culture of and norms 
participating in improve knowledge- for collaboration and 
knowledge building building discourse participation 
Improved skills 
Experience gained 
Rules Community Division of Labor 
Teacher’s expectations The participating students Students’ work 
Principles of good notes The course teacher Support from the teacher 


Traditional teacher-centered 


school culture and norms 
Tensions 


(a) Attaining object despite relatively low motivation and limited skills 
(b) Attaining object despite inadequate tools 

(c) Attaining object despite traditional teacher-centered school culture 
and associated norms for participation 


Figure 1. Activity System A: Before the KCA Student Activity 


After the students engaged in reflective assessment using the KCA, the dynamics of the student activity 
systems changed: the students subsequently attempted to advance their knowledge-building discourse by 
carrying out KCA-based reflective assessment. Figure 2 shows the activity system at this stage (activity 
system B), a nested system incorporating the outcome of the earlier activity system (activity system A) 
(Yamagata-Lynch, 2010). In activity system B, the KCA was introduced to help the students engage in 
reflective assessment. They reflected on the KCA data and further synthesized and rose above the 
community’s ideas. Initially, however, the students were unable to interpret the KCA data productively, 
leading to tension between the students’ efforts to use the KCA data to improve their discourse and their 
ability to interpret the data. However, this tension lessened as the students gradually acquired the skills 
needed to interpret and use the KCA data and synthesize their ideas. Their skills development benefited from 
frequent and opportunistic knowledge-building talks promoting reflection on and thoughtful use of the KCA 
data, with an overarching emphasis on discourse improvement as a collective responsibility. For example, the 
following excerpt from the KCA reflection sheet demonstrates how the student was able to generate synthesis 
and rise above ideas collectively through reflective assessment with the KCA data, from the question “Are 
we putting our knowledge together?”: 
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We are wondering whether we have really put our knowledge together by referring to each other’ notes, 
improved our understanding, and drawn conclusions...Look at the chart (pointing to the pie chart 
generated by the KCA), only 3% of our notes contain references... kind of very low...only 6% of our 
notes were used as references... You see, many of the summary notes are not quite good. Actually, we can 
further improve them (reading the pooled summary and rise-above note in the KCA)...we need to 
comment the notes referred to, for example why we choose them, why they are good or not...we need to 
refer more notes, summarize what they have talked about and identify further inquiry areas (from 
collective reflection in small groups) 


In this excerpt, the students carried out KCA-supported reflective assessment with the purpose of 
inquiring (“whether we have really put our knowledge together by referring to each other’ notes, improved 
our understanding, and drawn conclusions”), and analyze the possible issues (“only 3% of our notes contain 
references;” “kind of very low;” “only 6% of our notes were used as references;” “see, many of the summary 
notes are not quite good’’). Based on their analysis, the group of students appeared to reflect on the quality of 
reference notes, made a plan (“further improve them’) and took actions to address the issues (“we need to 
comment the notes referred to, for example why we choose them, why they are good or not...we need to 
refer more notes, summarize what they have talked about and identify further inquiry areas”). This example 
suggests that KCA-enhanced reflective assessment helped this student to connect their learning orientation to 
the knowledge-building goal of idea improvement as collective responsibility. 

During this process, the students also became more motivated. As a result, tensions (a) and (b) in activity 
system A (the need to attain an object with limited skills and motivation, and the need to attain an object in 
the absence of a community-oriented framework for data-driven idea improvement) were substantially 
alleviated. In addition, during the KCA-driven reflective assessment, a range of tools 
(e.g., knowledge-building talks, a community-oriented framework for data-driven idea formation, 
collaborative concept maps, and strategies for developing collaboration skills and increasing motivation) 
helped the students to develop a knowledge-building culture. As a result, tension (c) in activity system A 
(the need to attain an object in the absence of a knowledge-building culture and associated norms for 
participation) was also substantially alleviated. 

In activity system B, however, the teacher’s expectations clashed with the students’ limited attention, 
energy, and time. In addition, the expectations of the teacher and the research team clashed with students’ 
attention, energy, and time. Tension (a) arose from the students’ competing obligation to actualize the 
expectations of the teacher and research team and to complete their SBA projects. This tension is represented 
in a circular fashion because elements of the rule component created further tension in the system. 
Nevertheless, the overall outcome of activity system B was positive: the whole class collaboratively 
advanced their understanding by engaging in reflective assessment using KCA. Many of the students 
produced high-quality synthesis/rise-above notes, although others did not. The students continued to advance 
their collective understanding by gaining the skills, interests, and motivation necessary to pursue knowledge 
building. 
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Tools 
Increased motivation, skills, and experience and improved 
online discourse 
KCA and corresponding prompt sheets 
Knowledge Forum® and its integrated assessment tools 
Collaborative conceptmaps 
Knowledge-building talks promoting collaborative 
reflection on and thoughtful use of the KCA data 
Community-oriented framework for data-driven idea 


improvement 
Activity System A 5 
. . utcome 
suble ee : Object .,. Collective knowledge 
Students participating in Improving community’s 
; : advancement 
reflective assessment discourse through 


Increased motivation and interest 
Skills and experiences gained 
Good and bad synthesis and rise- 
above notes 


using the KCA reflective assessmenf— > 


using the KCA 


(@) 
Rules Community Division of Labor 
Teacher’s expectations Participating students Students advancing discourse 
Completing SBA projects Course teacher Support from the teacher 
Collaborative knowledge- Research team Support from the research team 


building culture and norms 
Tensions 
(a) Actualizing the teacher’s expectations while competing SBA projects 


Figure 2. Activity System B: After the KCA student activity 


Figure 3 shows Activity System C in which the knowledge-building participants are represented as 
subjects, and the intended collaborative improvement to the students’ knowledge-building discourse is 
represented as the object. Capitalizing on the outcomes of activity system B, the students were able to 
collaboratively improve their knowledge-building discourse and develop skills in areas such as 
summarization, inquiry, collaboration, and metacognition. Many of the students also became interested in 
knowledge building, synthesizing notes, and analyzing their conceptual progress. 

Tools 
Knowledge Forum® 
Online discourse 
Skills gained; increased motivation and interest 
KCA data and accompanying prompt sheets 


Activity System B 


Subjects Object Outcome 
Students participating in Improving knowledge- Improved discourse 
knowledge building building discourse and Improved skills and 

creating synthesis/rise- increased motivation 
above notes Synthesis notes 


Rules — Community Division of Labor 
Teacher’s expectations Participating students Students contributing notes 
Collaborative knowledge- Course teacher Support from teacher 


building culture and norms 


Figure 3. Activity System C: Immediately after the KCA Student Activity 
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4.2 Findings of the ASA 


The ASA yielded three main findings that explain the nature, dynamics and tensions of the social practices 
that arose from the students’ KCA-afforded self-directed reflective assessment. These findings offer insights 
into the elements that became influential cultural tools, and the ongoing activities that supported the 
transformation of these newly introduced artifacts into cultural tools. 

Regular tasks promoting inquiry, collaboration, and reflection introduced new tools that mediated 
student activities. Regular tasks were designed to help the students to develop skills in inquiry, collaboration, 
and reflection, and to create a collaborative culture and norms. Comparison of activity system A with activity 
system B revealed that these tasks constituted valuable tools mediating the students’ activities. For example, 
the students exhibited limited cognitive skills (inquiry and summarization) and metacognitive skills 
(reflection) in activity system A, and seldom reflected on their existing (lack of) knowledge (based on 
informal individual student interviews at the beginning of the course). Therefore, the teacher encouraged the 
students to write weekly reflections and create individual paper portfolios with the help of a task sheet. These 
regular tasks enabled the students to gradually develop cognitive and metacognitive skills that mediated their 
subsequent activities. Writing paper portfolios cultivated the ability to summarize information, which in turn 
mediated the students’ engagement in activity system B: some wrote synthesis notes in the style used to 
prepare the paper portfolios, i.e., identifying a string of good notes and explaining why these notes were 
good. 

Framing discourse improvement as a collective responsibility acted as new tools that mediated new 
activities. As indicated in activity system A and activity system B, framing discourse improvement as a 
collective responsibility helped the students to understand over time that knowledge building requires a 
collective effort to improve ideas. A knowledge wall was used to frame discourse improvement as a 
collective responsibility before the students began working in the Knowledge Forum (see activity system A). 
After they were introduced to the KCA, the students were guided to reflect on the assessment data with an 
emphasis on data-driven discourse improvement as a collective responsibility. The students were encouraged 
to help each other to make sense of the data, engage in joint action planning, and discuss and share strategies 
for improving the community’s discourse. The benefits of these activities became evident in activity systems 
B and C, as the students collectively advanced the community’s discourse by capitalizing on their collective 
reflections of the assessment data. Framing data-driven discourse improvement as a collective responsibility 
facilitated the creation of norms and fostered a sharing culture underpinned by a shared understanding of the 
processes and purposes of data use purposes and processes, and also stimulated student conversations about 
the data that improved the accuracy of their interpretations. Fostering norms and a culture that promotes 
critical inquiry, reflection, and the examination of underlying beliefs enables discourse to be improved 
through data use (Schunk & Zimmerman, 1997). 

New skills and increased interest were transformed into new tools that mediated the students’ new 
activities. As indicated in activity system B and activity system C, the students used their newly gained skills 
and interests to engage in new activities that allowed them to progressively advance ideas in the communal 
space. For instance, in activity system B, the students used the collaboration skills learned in activity system 
A (through collaborative concept-mapping and collaborative knowledge-wall building) to engage in note 
writing in the Knowledge Forum and reflective assessment. The knowledge-building process required the 
students to collectively advance ideas and help each other to improve their notes. Combined with the skills 
gained in activity system A such as summarization, inquiry, collaboration, and metacognition, the students 
gradually exhibited increasing engagement in reflective assessment through the KCA and the production of 
knowledge-building discourse. 


5. CONCLUSION 


In this study, we found that various components of the activity systems shaped the students’ 
knowledge-building behavior and reflective assessment in the KCA. These components were the skills 
required for data use, the establishment of a collaborative culture and norms for participation, the 
opportunities available for collaborative reflection about data, and the framing of data-driven discourse 
improvement in terms of collective responsibility. Our findings on the interactions between the conditions of 
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and tensions within an activity system, and the processes by which such factors become either affordances or 
constraints to students’ collaborative use of data, suggest ways of structuring change in classrooms and even 
schools, particularly to meet the needs of educationally disadvantaged students. Our findings also lay the 
groundwork for future research on students’ collaborative work and metacognitive activities in relation to 
data use. 

Our findings indicate that the KCA and its accompanying prompt sheets can help students to engage in 
productive reflective assessment and focus on the key learning goals of knowledge building, thereby 
collectively improving the discourse created by a community. Also, the findings suggest a design with three 
important components that are accessible and feasible for low-achievers: (a) establishing a collaborative 
culture and norms for participation, practicing cognitive skills such as summarization and concept mapping, 
(b) periodic tasks that promote collaborative reflection and thoughtful use of the KCA, and (c) framing the 
data-driven improvement of discourse as a collective responsibility (Scardmalia, 2002). 

Within knowledge-building research, studies have mainly relied on online discourses and quantitative 
server-log data in the Knowledge Forum database. Very few studies have made use of qualitative data such 
as interview transcripts, student artifacts, and face-to-face teacher-student and student-student discourse to 
characterize the social practices that arise from knowledge building. However, to gain a fuller understanding 
of knowledge building, it is necessary to understand the relationship between online discourse and the nature 
and dynamics of the social practices that develop during knowledge building (Zhang et al., 2007). This study 
has addressed this gap in the literature. 
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